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ABSTRACT 

This study seeks to illuminate those characteristics 
of students which predict their performance in using evidence to test 
a hypothesis. The research is conceived as part of a larger 
investigative strategy for understanding the relationships among 
characteristics of students^ characteristics of social studies 
thinking tasks^ and characteristics of instructional variables. 
Stepwise multiple regression using five variables— verbal ability^ 
age^ sociceccnomic i>tatus^ educational expectation^ and sex — is 
employed to predict students' use of evidence. Subjects are 304 
students in grades seven, nine, and eleven. Results reconfirm the 
importance of verbal ability as a predictor in verbally loaded tasks 
(predicting 25 percent of variance), but surprisingly neither age nor 
socioeconomic status are found as significant predictors. 
Implications are drawn for curriculum decision making. Suggestions 
for further research are enumerated. (Author) 
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UU AN ASPECT OF CRITICAL THINKING: PREDICTING STUDENTS' 

USE OF EVIDENCE 



Ted Feely 

UNIVERSITY OF DELAWARE 
ABSTRACT 



This study seeks to illuminate characteristics of students which 
enable us to predict their performance in using evidence tc test a 
hypothesis. The research is conceived as part of a larger investigative 
strategy for understanding the relationships among characteristics of 
students, characteristics of social studies thinking tasks and characteristics 
of instructional variab'^es. Stepwise multiple regression using the variables, 
Verbal Ability, Age, Socio-Economic Status, Educational Expectation and 
Sex is employed to predict students' Use of Evidence, Subjects are three 
hundred and four students in grades seven, nine and eleven. Results re- 
confirm the importance of verbal ability as a predictor in verbally 
loaded tasks (predicting 25% of variance), but surprisingly neither Age 
nor Socio-Economic Status are found as significant predictors. Implica- 
tions are drax-m for curriculum decision making. Suggestions for further 
research are enumerated. 
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AN ASPECT OF CRITICAL THINKING: 
Predicting Students' Use of Evidence 



OBJECTIVES 



Trends and fads have come and gone, but for at least the past thirty years 
there has been a recurrent theme in the social studies literature that students 
must be taught to be independent, critical thinkers, KTiether ve have used the 
term "problem solving," "reflective thinking," "critical thinking," "inductive 
approach," "discovery," or "inquiry," we have meant — at least in part — that 
students should learn to think on their oun. 

This study seeks to illuminate the characteristics of students which enable 
us to predict their ibility to make use of evidence in testing statements, "'he 
study extends research reported last year on the performance of 9th grade students 
on some critical thinking tasks • That research identified verbal ability as the 
single most robust variabl^e in the prediction of student performance. This study 
extends the earlier research by introducing several other variables which might 
be expected to contribute to a further understanding of students' critical thinking. 
These variables are age, socio-economic status, educational expectation and sex. 

THEORETICAL FRAMEWORK 



As the term is used here, critical thinking is defined as the evaluation of 
evidence or argument in the light of acceptable standards for the purpose of 
deciding v/hethe; to accept or reject a statement. This definition is derived from 
the works of Smith and Ennis. (Smith, 1953), Cinnis, 1967). Building from Ennis' 
analysis, critical thinking is operationalized as a set of tasks v;hich call upon 
students tD engage in critical thinking operations. Just as arithmetic deals with 
tasks which require the operations of combination, division, etc., criticr.l thinking 
in the social studies can be conceived as a group of tasks which require operations 
such as judging the relevance of evidence, judging the reliability of evidence, 
judging whether a statement follows from premises, identifying unstated assumptions, 
etc ♦ 

The research is based upon the hunch that (1) characteristics of students and 
(2) characteristics of tasks with v/hich students are expected to interact should 
prove to be a fruitful line of invest gation . This study is part of a larger research 
strategy which is seeking to establish a baseline of information about the relation- 
ships ar^'ng students* v-^haracteristics and social studies thinking tasks. The base- 
line of information shouJd, then, enable a systematic investigation of instructional 
variables and their impact upon student performance. In the meantime, it is 
expected that results of the present study plus related studies should provide use- 
ful knowledge for helping us understand our students' performance in critical thinking. 

The major question being asked in this study is whether there are identifiable 
aptitudes and characteristics s^-id^nts which enable us to better understand their 
use of evidence in testing statements, or hypotheses. More specifically, this study 
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examines five characteristics of students as predictors of performance in using 
evidence: students' Verbal Ability, Age, Socio-Economic Status (SES), Educational 
Expectation and Sex. For each of these characteristics, this study asks the question, 
"Does this characteristic of students enable us to predict performance in using 
evide.ice?" 



>ffiTHOD 
Data Analysis 



Stepwise multivariate regression analysis was performed on the data using the 
Statistic a-' Packa^ fc)C the Social Sciences (SPSS) programs. Stepwise regression 
enables the entry of predictor variables into the analysis in a manner such that 
each predictor variable operates as a covariant, or control variable, for each 
successive predictor variable. 



Statistical Significance 

Two criteria were employed to establish significant results. Statistical 
significance was set at .05. More importantly, no predictor sliaring less than 
5% of the total ^^riance^^^^j^the criterion variable was considered a significant result. 

Variables 

Criterion Variable — 

Students* use of evidence in teotmg a statement was the criterion, or dependent 
variable. This was measured by administering Sec^ion I, Part A of "The Cornell 
Critical Thinking Test, Level X." This part of the test is comprised of twenty- 
three items plus two practice items. Reliability of the instrument for all classes 
using the KR20 procedure Gf inter-item reliability was .754. The test was administered 
by teachers in their social studies classes. This section and part of the Cornell 
Level X Test measures students* perfOxTinance in distinguishing evidence which is 
supportive, negative or irrelevant to a statement. 

Predictor Variables — 

Five variables were used to predict student use of evidence: verbal ability, 
sex, age, educational expectation and socio-economic status. 

Verbal ability was measured using the "Wide Range Vocabulary Test" with the 
format altered to permit machine scoring. This instrument proved highly reliable. 

Age was measured by self-reports of students and calculated in months. Students 
came from 7th, 9th and 11th grade classes. Although age is not itself an explana- 
tory construct it is associated with general mental maturity and in the case of 
several developmental theories, notably that of Piaget, it is a consistent correlate 
of certain logical operations. Peel identifies mid-adolescence as a time when 
students demonstrate an ability to make co-ordinated judgments in the fields of 
history and geography. 
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Both sex and SKS, \^»ere measured by self-reports. SES \v»as indicated by a 
singJe raeasure of students ol their father's occupation. This self-report x^as then 
coded by a single coder employing a scale developed by N.O.R.C. Of all the measures, 
SES is the least satisfactory. 

Educational exp-jc t ation x-ias generated by asking students to indicate the highest 
schooling they expect to conplete. This is a measure which might well be expected 
to correlate higiily with SES and wrbal ability but which is included as a rough 
measure of students' perception of the place of schooling in their future. 



SUBJECTS 



Three-hundred-four students from two schools took part in this research. The 
schools v/ere from a single school district in a ^rov/ing small city of about 150,000 
population in the Middle ^^tlantic States region. The research made use of \>;hole -class 
groups vjhich were chosen provide a wide range of academic ability. There were 
113 7th grade pupils, 94 9th grade pupils and 97 11th grade pupils. The pupils were 
overwhelmingly white v;ith blacks comprising no more than 10 percent of the sample. 



RESULTS 



Eefore examining results of the stepwise regression analysis, we will take a 
look at some descriptive data which should help us ''get a feel" for the data. Table I 
displays the correlation u.^trix among the predictor and criterion variables. 



Table One about Here 



The results which are of most interest are found in the column farthest to the 
right in which each of the predictor variables is correlated with the criterion 
variable. With the exception of Sex, all of the predictors are correlated positively 
and significantly with Use of Evidence. 

That is, with the exception of Sex, all of the variables \7avy in approximately 
the same manner as Use of Evidence. For example, the correlation of .39 betv/een 
Age and Use of Evidence indicates that as Age goes up performance on Use of Evidence 
tends to go up, etc. 

It must also be noted that the predictors correlated with one another. For 
instance, the largest coefficient of correlation is between Age and Verbal Ability. 
There is also a statistically significant relationship among Verbal Ability and Socio- 
Economic Status. Thus, the important question which remains is how the mix of variables 
contributes to prediction ol students' use of evidence. To examine this, stepwise 
multiple regression is used. 

Results are summarised on Table 2. Verbal Ability was entered as the first 
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oredictot. Thus, it served as a covariant for all later variables. Other variables 
ll~ec..l au^caticaily by the progra. in the order of their -f.^h^.i: s^rrf 
The table below provides five values for each variable. B and Beta values are 
the regular and normalized regression coefficients, respect ve y. The ^^^^l' jf- 

s the'standard error of B. Where this value is considerably less f^f." J'^f^.f^'^^^^, 
sign of the regression coefficient can be considered signiicant F is an ^"d.cator 
of statistical significance. In this case, any F value falling between 2.25 and 
3 10 has a 5% probab.litv of chance. Any F value falling at 3.11 or above has a 
1% probabil y of ch.mce'. The last column on the table indicates the percentage of 
variance predated by the variable. As indicated in the earlier discussion of 
"Statistical Significance," this last value on the table is of ^-^f ^^^^^^ .^ct 
quite possible for a variable to be statistically significant, but for it to predict 
so little variance in the criterion variable that it is of no practical importance. 



Table Two About Here 



Verbal Ability, the first predictor to enter the analysis shows botn a b gh 
statistical significance with a chance probability of less than •OOl/'^'^^^^^^^^f !^ 
significant prediction of variance with .:6 percent. That is, students' Verbal Ability 
predicts about ona-fourth of their perfo.mance on the Use o Evidence measue. The 
F ratio for Sex indicates that the findings are statistically significant less 

h r 01 ^hanc: JesuUs); however, it predicts only 1 2 P-f-^/f.^^J^.^-J^:^: °" 
the Use of Evidence. The negative si^ns in front of the B and Beta 

ndicate that females performed bette: than males. As with Sex Age ^^^^/^^^'^^^^^^ 
significant (less than .01 chance results), but contributes only 19 percent of 
variance! Finally, only Socio-Economic does not. contribute a statistically signifi- 
cant prediction. 

several results are notable. Verbal Ability contributes a f ^^J^^ /"^fJ^Se 
of variance in the prediction. This result i3 certainly ^^P^^^^^^/^J^^^ ""i:;'bre 
with both theorv and previous results, i^liat is most surprising is the negligible 
^^n^r uaon of Age and SES to the prediction. Age has more than a modest correla- 
tion ( 39) with the criterion variable, but that relationship relies almost 
thrall; u on verbal Ability. The weak predictive P^"-^ .^f3/^^ ^ fe^er 
explain. Its low correlation with the criterion variable (.15) leads to several 

possible explanations. 

The first potential explanation would be that subjects in "^^e study were 
relatively homogeneous with regard to SES and, therefore, this variable had little 
correlational or predictive power. The Socio-Economic Index scale used in this 
studv has a range of values from 0 to 96. Seventh graders, ninth graders and 

e enfh gra ers in this study had a range of values of 86, 92 and ^2 respec ively^ 
The stindfrd deviations of values were 22, 23 and 23 respectively. Thus, this first 
potential explanation is not supported by the data. 

The second potential explanation is that the values are not reliable. There 
are at least ^wo potential sources of unreliability. The first flows from the 
are at leasc lwu i^ulc.il ,„„invf.ri That is students were asked to report 

O fact that a single measure was employed. mat is, bcuucn 
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their father's occupation. The second flows from the fact that a single rater 
translated reports of occupation into SCS values. To check this source of unrelia- 
bility a second rater, operating independentl> rated occupations and inter-rater 
reliability of .90 was calculated. 

Of the remaining variables, both Sex (favoiing females) and Educational 
Expectation proved statistically significant; lnwever, neither contributed a 
sufficient percentage of variance to be considered a significant predictor. 

llius of the five predictors of students* Use of Evidence, only Verbal Ability 
contributed significantly to preoiction. Interactions? among each pair of predictors 
and the criterion were tested ana found to add nothing to the prediction. 



I>Q^LICATIONS AND FURTHER RESEARCH 
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The most surprising result of this study was the paltry relationship between 
the SES indicator and the measure of students' use of evidence. Given the consistent 
findings of strong relationships between SES and school achievement, (most notably 
in the "Coleman Report") I expected SES to add significantly to the prediction of 
students' performance in the use of ev:.dencc. Wnether the result is a solid 
finding or simply the result of error xn measurement or some unkno^^fn peculiarity of 
the sample awaits further study. 

In terms of curriculum decision making, the nost significant findings are the 
relatively high predictive power of Verbal Ability vpredicting 26% of variance) and 
the almost non-existent predictive power of Age (about 2%) when Verbal Ability is 
the covariant. (Age predicts about 12.8% of variance when Verbal Ability is not 
used as a covariant.) These findings downplay the importance of age—and its cor- 
relate, maturation— as an im.portant variable in explaining students' Use of Evidence. 
More specifically, these findings indicate that Verbal Ability is a more important 
predictor than Age of how well students will be able to interact with a curriculum 
that calls upon Use of Evidence. 

It is important to note that this research does no^ purport to identify any 
fixed and inimutable level of Verbal Ability oolow which students are incapable of 
using evidence. Indeed, it m.ay be possible to identify instructional or curriculum 
practices wnich can help students overcome weaknesses in ability to use evidence. 
However, the research findings indicate that if a social studies department is inte- 
rested in matching an evidentially based curriculum with students who can perform 
well in that curriculum, it is better to use Verbal Ability than Age as a criterion. 

The limitations within which it is reasonable to interpret results of this 
study point toward f'orther potentially profitable studies. First, it should be 
noted that this study operationalized the construct "Use of Evidence" in a very 
specific way. The task given the students was verbal and given in a written format. 
It dealt with what is for most people a relatively familiar set of meanings. That 
is, it dealt with the concept of death and the implications of death— e.g., that one 
is not around to disturb things, etc.— In short, all of these characteristics of the 
task might be expected to influence students' performance. Further studies might 
wish to operationaiize the task in a different manner. In particular, it would-be 
informative to develop several additional instruments which follow the format of 
the Cornell instrument but which deai with conceptual matter of differing levels 
9^.of difficulty. 
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Second, it must be noted Khat this research examined 12 to 16 year olds and 
that results need to be interpreted in this light. Further research niigl.t: 
examine performance of youngor students in using evidence. The difficult} is, of 
course, in developing instruments which are sensitive to the subjects. IL might be 
useful to develop visual displays of evidence and oral i. terview instruments. 

Third, further studies should examine predictors of student performance on other 
critical thinking tasks. (The author is currently examining student ability to 
distinguish between causative and correlative explanations.) Fourth, it should 
be noted that the five predictors employed in this research were able to account 
for only about 31 percent of the shared variance among the Dredictors and the 
criterion variable. it would be very useful to know if there are any further 
predictors which enable us to enhance prediction. Increasing the reliability of 
instruments shoula increase the percentage of variance accounted for. 

Fi*"th, a more rigorous examination of the predictive ability of SES needs to 
be undertaken.: Multiple measures of SES should be included. Sixth, it should be 
profitable to examine students' performance on critical thinking tasks in the light 
of Piaget's work. If Piaget is correct in his identif iCotion of logical develop- 
ment through stages which are largely independent of Verbal Ability, it should be 
possible to identify at least some critical thinking tasks in which Piagetian 
indicators predict a significant amount of variance beyond that predicted by 
Verbal Ability. Finally, it should be remembered that replication is the best 
test of the reliabilicy of research findings. 
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TABLE 1 



Correlation Matrix Among All Variables 



Use of 

Age V*A. SES Ed. Ex. Sex Evidence 



Age 1.00 .54** .03 -.11* .02 .39** 

Verbal 

Ability 1.00 .23** .17** .11* .51** 

SES 1.00 .17** .04 .15** 

Educational 

Expectation 1.00 -.05 .19** 

Sex 1.00 -.07 
Use of 

Evidence 1.00 



p C -05* 
p < .01** 



er|c i 0 



TABLE 2 



Multiple Regression Analysis: Predicting Students' Use of Evidence 



Variable 



Std. Error 
Beta of B 



% of Variance 
Predicted 



Verbal Ability 

Sex 

Age 

Educational 
Expectation 

Socio-Economic 
Status 



.087 
.819 
.035 

.260 



.396 
-.111 
.189 

.134 



0.016 
0.423 
0.013 



31.01*** 
3.75** 
7.31** 



0.116 5.01** 



-0.015 -0.043 0.020 Insig. 



26.0 
1.2 
1.7 

1.9 

Unreliable 



P < .05* 
P < .01** 
P <. .001*** 

Interactions among all of the predictor variables were tested and 
found insignificant. 
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APPENDIX A 



i'r^i I, Section A, VlMt M 'P'hmv c l o t lu- '^l 'A > 'oup 

The j'irst job of your p,roup i lo 'ind out \'IkU luippoiu thv^ iirst [;roup of 

15 cxploivM-s. Y'^r.r :;:*oup Innck'ci on iacon.i and has ]usi (i i ^ o-\toc1 lIk i.ioL^l lu:t.s 

put up by the irr»t '.roup. Vac huts .vpponr to he in .od condiLi-n ["roin the outside. 
It is a \Mrn day and the sun is -.hinin;. *lhv' lues, looks, , and birds make this 

part ol i:w:on5a appear about like parts of ct^ntral United Stato^. 

You and the health officer are the first to arrivo at the group of huts. You 
call out, but got no ansv;er. 

The heal th. officer sugp;ests, "Maybe they're all dead." You investigate. 

Belo\; are listed a numl r of facts \;hich you learn. For each fact you must decide 
if it \-ould bo evidence for, or evidence against, tht. ilea 1th ofTicer's idea that t hey 
.n-t* all dead . It^vover , the fact niight not be neitiier evidence lor, nor evidence 
again.sC the iH'altii oificor's idea. 

For each fact, nark one of the folloiJing on your answer sheet: 

A. This fact is evidence in support of the health officer's idea that they 
are all dead. 

B. This fact is evidence against the health officer's idea t.hot they are 
all dead. 

C. Neither . Discovery of this fact makes no difference. 
Here is a sample: 

1. You go into the first hut, !■ very thing is covered by n thick layer of dust. 

This fact is evidence in suppo: t of the health officer's idea. It certainly is 
not enough to prove his idea, but is evidence for it. Using the special pencil, 
blacken the .space under A for No, 1 on your answer sheet--like th.is: 

ABC 

^ :: :: 

Here is another example: 

2, Other members of your group discover the first, group's rocket ship nearby, 

Uhat do you say about that fact? Mark your answer by No, 2 on your answer s!:e-, l. 
Do it now • 

You should have marked C. This fact about the rocket shp does not help us decide 
\;hether the members of the first group are dead. If you did not mark C, erase your 
mark thoroughly and mark C. 

Hero is a list of facts.^ For each one mark A, B, or C. If you have no idea 
\-;hich to mark, leave that one blank and go on to the next on^ . Consider the bearing 
of each fact at the t i re that it becomes known . Do so in thj o/der in ^?hich the v are 
numbered . I.'ork Glo\;ly and carefully, and do not return to r.n item once vou liave left 
it. Reminder- -mark as follo\;.5: 



CO ON TO TWK UEXT PACr. 



